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Abstract: Let X be a statistical variable representing student ratings of University teaching. It is 
natural to assume for X an ordinal scale consisting of k categories (in ascending order of 
satisfaction). At first glance, student ratings can be summarized by a location index (such as 
the mode or the median of X) associated with a convenient measure of ordinal dispersion. For 
instance, the median of X may be associated with the dispersion index of Leti, resulting in a 
synthesis that takes into account the ordinal nature of data and also communicates 
information in an effective way. More generally, there are many indexes (such as the ordinal 
entropy) that can be properly employed to measure the ordinal dispersion. On the other hand, 
student ratings are often converted into scores and treated as a quantitative variable. More 
generally, it is possible to measure student satisfaction by means of a real-valued function 
defined on the standard simplex and satisfying some appropriate conditions. Finally, such a 
measure of satisfaction can be associated with a suitable measure of variability. 
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1. Introduction 

The evaluation of University teaching by students is generally carried out by 
administering a questionnaire. Such a questionnaire requires responses on an ordinal scale 
consisting of k categories m 1 < ... < m k in ascending order of satisfaction. 

For example, the question "Overall, are you satisfied with this course?" (in the 
questionnaire adopted by Italian Universities) can be answered by "Decidedly no", "More no 
than yes", "More yes than no" or "Decidedly yes" (k — 4). 

Focusing on responses to a single item in the questionnaire, the issue immediately 
arises of synthesizing the corresponding distribution of relative frequencies (shown in Table 
1) by appropriate measures of location and dispersion. 
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Table 1 . Distribution of relative frequencies. 


Ordinal categories 

m, 


m, 


Relative frequencies 

Pi 


Pi 



The discussion of this issue, introduced in Section 2, is developed in Sections 3, while 
Section 4 extends the study to multidimensional measures. 

2. Location and dispersion of ratings 


In order to take into account the ordinal nature of student ratings, it is appropriate to 
represent the standard simplex of frequency distributions with k categories 



( 1 ) 


as A' k = : 1 = > F' 2 > ... > F[ > 0] using the relative inverse cumulative 


k 

frequencies (/ = l,...,k). It is also appropriate to require that a location index 



preserve the stochastic ordering, i.e. if 



with 



then 



Such an index, in fact, reaches its minimum or maximum value, respectively, in the 
case of complete dissatisfaction (p, — 1) or total satisfaction (p k =1): 



In addition, such an index of location, as the median me = min{m ; : F' +1 < 0.5}, can 
be associated with a suitable measure of ordinal dispersion, as the index of Leti 


k 



(3) 


or the ordinal entropy 
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k 


=-Z[ F . l0 8( F .) +F >*( F /)] e [°'( t - 1 ) 1 0g(2)] (4) 


These indexes allow to decompose the overall dispersion measured in a set of C University 


courses in two parts, namely "between" and "within" the courses: 

k -1 


G, J =2Y,FXl-F,)=G w „ ll , I+ G ! 


within 


between 


(Grilli and Rampichini, 2002) and 


k -1 


F> )] FI within FI between 


i=l 


where 



c =1 /=1 



C k -1 

nothin = "2X Z ^ lo gfe )+ l 1 - F i\c J’ogO - F i\c )] 


within 



i C 

and F j = ’^ j p h = V' K c F. (/ = is the relative cumulative frequency expressed as a 


c 


c=1 


finite mixture of relative cumulative frequencies F,| c , corresponding to ratings of the course c, 
with weights 7Z C given by the proportions of students who evaluate the course c (c = 

In this regard, we note that the higher is the value of the "between" component, the more is 
the influence of the course features on the student ratings. 

In general, an index of ordinal dispersion D attains its minimum if 


3/ < k p i = 1 

and attains its maximum in the case of polarization 


Pi=i = Pk 


(Leti, 1983). This happens for the indexes (3) and (4). 

In particular, it follows that H ord > G ord and the index (4), when compared with (3), gives 
more weight to the frequencies 



when they are small. 
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In the case of four courses and k — 4, Table 2 illustrates how the above approach 
produces results easy to communicate, despite being characterized by a lack of ability to 
discriminate situations very different from each other, because of the poor range in (2). 


Table 2. Some traditional measures of location and ordinal dispersion in the case of k — 4. 



m, 

m 2 

m 3 

m 4 

Location 

Dispersion 

Course 

Pi 

Pi 

Pa 

P4 

Moc/e 

Median 

Leti 

Entropy 

1 

0.10 

0.20 

0.30 

0.40 

m 4 

m 3 

1.08 

1.61 

2 

0.01 

0.20 

0.39 

0.40 

m 4 

m 3 

0.83 

1.24 

3 

0.01 

0.10 

0.49 

0.40 

m 3 

m 3 

0.70 

1.08 

4 

0.10 

0.30 

0.30 

0.30 

nd 

m 3 

1.08 

1.61 


Table 2 also shows that the mode, as location index, does not preserve the stochastic 
ordering. 


3. Satisfaction indexes 

From a general point of view, a satisfaction index can be identified with a real¬ 
valued function defined on the standard simplex (1) 

5 : A* -> SR (5) 

for which the following two conditions apply. 

Condition of strict monotony 

The function S is said to be strictly monotone if it is strictly increasing with respect to 
the stochastic ordering, that is to say 


V/ < k Vj < j \/p e \),p] 

S(p x ,...,p i ,...,p j ,...,p k )< S(p l ,...,p i -p,...,p j +p,...,p k ) 


( 6 ) 


Being a reinforcement of stochastic order preserving condition, (6) implies the 
following condition 




[ min(S'): p x = 1 
[max(S '): p k =\ 


(7) 


Condition of equivariance 

The function (5) is said to be equivariant if it is equivariant with respect to the order reversing 
permutation of i K p { ,p 2 ,...,p k _ l ,p k ), that is 

S(Pi » PiPk-i,Pt)~ min ( 5 ) = max ( S )~ S (Pk > Pk-v-’Pi’Pi)- ( 8 ) 
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This condition identifies the value s of (5) corresponding to any frequency distribution 
invariant under order reversing permutation, i.e. such that 



since from (8) follows immediately s = [min(,S) + max(5')]/2 , indicating a medium level of 
satisfaction. 


The following are some examples of satisfaction indexes: 


CG = 2p ] +5 p 2 +1 p i +10 p ^ 

(k — 4), due to Chiandotto et al. (2000) and used in teaching evaluation by Universities of 


Cagliari, Ferrara, Florence, Macerata, Milan, Palermo, Sannio Sassari, Trieste and Urbino; 


M u =~p, -up 2 + Up 3 +p 4 


(k = 4, 0 < u < 1), studied by Marasini et al. (2011) and used by Universities of Pavia and 
Insubria; 


k 


k 


sdi =y,f; =i>. 


proposed by Cerchiello et al. (2010) and employed by Universities of Bergamo, Brescia, 
Naples East, Molise, Rome - Tor Vergata, Turin, Venice - Ca 'Foscari and Verona. 
However, the index due to Capursi et al. (2001) 



(r > 0) is strictly monotone, but is not equivariant, except in the case r = 1; while the index 
introduced by Civardi et al. (2006) at University of Milano - Bicocca 


r 


Pa P i 


Pa + Pa ~ Pi ~ Pi + v 


VPa + p, Pi + pj e j_ u ] 


CL 


V 


1 + V 


(k = 4, 0 < v < 1) meets the condition of equivariance and (7), but strict monotony is not 
met. 

In the case of four courses and k — 4, Table 3 illustrates the above considerations. For 
instance, the first course and the third course show that CI V , with v = 1, is not strictly 
monotone, whereas the first course and the last course show that the index 



is not equivariant (if this index were equivariant, then we would have 0.30 instead of 0.35 in 
the last row of the Table 3). 
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Table 3. Some popular measures of student satisfaction in the case of k = 4. 


Course 

Pi 

Pi 

Pi 

Pa 

CG 

M 0 .5 

SDI 

1^0.5 

ci, 

1 

0.10 

0.20 

0.30 

0.40 

7.30 

0.35 

3.00 

0.70 

0.32 

2 

0.00 

0.30 

0.30 

0.40 

7.60 

0.40 

3.10 

0.81 

0.49 

3 

0.10 

0.00 

0.50 

0.40 

7.70 

0.55 

3.20 

0.78 

0.12 

4 

0.40 

0.30 

0.20 

0.10 

4.70 

-0.35 

2.00 

0.35 

-0.32 


In general, the weighted mean 


=Tj w iPt 

i=i 

is a function defined on the standard simplex, which is strictly monotone, if it satisfies the 
inequalities 


W 1 < W 2 < ••• < W k -1 < W k ' ( 10 ) 

and equivariant, if it also satisfies the equalities 

w i -w l =w k: -w k+l _i (11) 

(i — 1 Thus the linear function (9) with weights satisfying conditions (10) and (11) can 

be called linear satisfaction index (being strictly monotone and equivariant). 

More specifically, each linear satisfaction index is completely determined by a weight vector 
W = {yv x ,...,W k ) satisfying (10) and (11), and sends the generic element p = (p l ,...,p k ) of 

k 

the standard simplex (1) to the point defined by the inner product p • w = e 

i=i 

Such a point, determined on the basis of the k coordinates p x ,...,p k , can also be 
represented as 

S w = 0 - S*K + S^w k = w, + S*(w k - Wi) 

by means of the coordinate 


w k - Wj 



defined through standardization. In this way, to every satisfaction index S w there correspond 
a standardized index 



W/A 


where the weights 


W: - W , 


W; = ' 


w t - W, 


k+l-i 


W, - Wy 


w k - Wj 


[ 0 , 1 ] 
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(/ = l,...,k) satisfy (10) and (11). Moreover, we can see that (11) takes the simple form 


It is interesting to note that some linear satisfaction indexes can be interpreted in 


terms of probability. To see this, let X and Y be random variables providing ratings of a 


student chosen at random, respectively, from the observed distribution p l ,...,p k and from a 
virtual distribution we require to be invariant under order reversing permutation. It is thus 


possible to define the probability that a student randomly selected from the observed 


distribution provides a rating not less than the virtual rating drawn from an invariant 
distribution corresponding to a medium level of satisfaction: 


k 


P(X>Y) = Y j P(Y<m)p : . 


i =1 


Similarly, we may define the difference P(X > Y) — P(X < Y). 


For instance, if we choose (y,,...,y) as the invariant distribution of Y, we get 


k 





and 


K2i-k-\ 2SDI -k-\ 
L —, —Pi = , 


P(X > Y) - P(X < Y) = £ 

i=\ k 


which for k — 4 becomes 



is a weighted mean that does not meet the conditions of strict monotony and equivariance, 
while 


P(X > Y)- P(X < Y) = - Pl -\p 2 + \p, + p A = M V2 . 


A further advantage of the linear satisfaction indexes is that we can measure the 
ordinal dispersion (in the sense of Section 2) by means of the variance 



which can be decomposed as 


k 


= X ( W ' - S - ) 2 Pi = D within + A 


within 


between f 
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where 


and 



c 

Pi = 2Xa-i c 

c =1 

(/ = 1,...,k) is the relative frequency expressed as a finite mixture of relative frequencies p,| c , 
corresponding to ratings of the course c, with weights given by the proportions of 

students who evaluate the course c, and S^ c is the linear satisfaction index for the course c 
(c = 1—C). 

4. Multidimensional satisfaction indexes 

If we extend our view to include C courses evaluated by a questionnaire with Q 
questions, we may construct a multidimensional index of the form 

c 

MS = 

c =1 

(Russo, 2002), being n c the proportions of students who evaluate the course c (c = 1,...,C) 
and 


MS C 



where, referring to the question q (q = about the course c, S and D cq represent, 

respectively, a standardized satisfaction index and a standardized ordinal dispersion index (in 
the sense of Section 2). 


5. Conclusions 

The paper shows how to measure student satisfaction through real-valued functions 
satisfying some appropriate conditions. In this context, further work will involve providing a 
complete characterization of satisfaction measures and a general representation of them. 
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